Selecting Among Heuristics by Solving Thresholded k-Armed Bandit Problems

Suppose we are given k randomized heuristics to use in solving a combinatorial problem. Each heuristic, when run, produces a solution with an associated quality or value. Given a budget of n runs, our goal is to allocate runs to the heuristics so as to maximize the number of sampled solutions whose value exceeds a specified threshold. For this special case of the classical k-armed bandit problem, we present a strategy with O( √ np∗k ln n) additive regret, where p∗ is the probability of sampling an above-threshold solution using the best single heuristic. We demonstrate the usefulness of our algorithm by using it to select among priority dispatching rules for the resource-constrained project scheduling problem with maximal time lags (RCPSP/max).